Online assessment standard setting for multiple choice questions.

Standard setting of assessment questions has matured as a psychometric practice since the 1980s1 and is now a common practice in Medical Education.2 This is done to ensure that a fair, defensible cutoff score or pass mark is reached for each assessment paper.3 Standard setting is thus a policy decision as standards are not a natural phenomenon waiting to be discovered,4 neither does the process signify scientific precision. Nevertheless, in order to be defensible standardsetting must be procedurally credible. At the College of Medicine and Health Sciences (CMHS), United Arab Emirates (UAE) University, the Modified Angoff5 procedure of standard setting has been adopted as applicable to all high stakes and end course MCQ examinations since 2014. This test-centered, criterion referenced method of standard setting is one of the most studied and widely used procedures for high stakes examinations.4 At the CMHS, following a period of question vetting, the procedure was initially conducted in face-to-face meetings of appropriate faculty members with subject expertise. In this exercise judges make independent estimates of the proportion of minimally competent candidates that would be expected to answer each item correctly. Where there is a large variation in estimates, judges then discussed discrepancies and revised estimates. Depending on length of discussion and necessity to refine previously vetted questions, the procedure often took an average of three to six hours to standard set an examination of 100 MCQs. In the case of high stakes examinations, the procedure could take several times that amount of time. Although this process had its benefits in further vetting of questions, useful discussion and feedback between faculty members about their understanding of the task, issues surrounding the minimally competent candidate, etc., it was also critiqued by faculty as arduous and time consuming for busy medical subject matter experts. Availability of new technologies have however facilitated evolution of methods and processes of test administration and setting cut scores to meet challenges posed by real and perceived inadequacies of existing processes.6,7 The purpose of this paper is to describe how we implemented an online standard setting procedure which significantly reduces the amount of time needed to standard set examinations.